Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retraining

A challenge of channel pruning is designing efficient and effective criteria to select channels to prune. A widely used criterion is minimal performance degeneration. To accurately evaluate the truth performance degeneration requires retraining the survived weights to convergence, which is prohibitively slow. Hence existing pruning methods use previous weights (without retraining) to evaluate the performance degeneration. However, we observe the loss changes differ significantly with and without retraining. It motivates us to develop a technique to evaluate true loss changes without retraining, with which channels to prune can be selected more reliably and confidently. We first derive a closed-form estimator of the true loss change per pruning mask change, using influence functions without retraining. Influence function which is from robust statistics reveals the impacts of a training sample on the model's prediction and is repurposed by us to assess impacts on true loss changes. We then show how to assess the importance of all channels simultaneously and develop a novel global channel pruning algorithm accordingly. We conduct extensive experiments to verify the effectiveness of the proposed algorithm. To the best of our knowledge, we are the first that shows evaluating true loss changes for pruning without retraining is possible. This finding will open up opportunities for a series of new paradigms to emerge that differ from existing pruning methods. The code is available at https://github.com/hrcheng1066/IFSO.Comment: chrome-extension://ogjibjphoadhljaoicdnjnmgokohngcc/assets/icon-50207e67.pn

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of seven pairs of contrast settings for pruning (e.g., unstructured/structured) and explore emerging topics, including post-training pruning, different levels of supervision for pruning, and broader applications (e.g., adversarial robustness) to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide some valuable recommendations on selecting pruning methods and prospect promising research directions. We build a repository at https://github.com/hrcheng1066/awesome-pruning

Demographic information prediction: a portrait of smartphone application Users

Demographic information is usually treated as private data (e.g., gender and age), but has been shown great values in personalized services, advertisement, behavior study and other aspects. In this paper, we propose a novel approach to make efficient demographic prediction based on smartphone application usage. Specifically, we firstly consider to characterize the data set by building a matrix to correlate users with types of categories from the log file of smartphone applications. Then, by considering the category-unbalance problem, we make use of the correlation between users’ demographic information and their requested Internet resources to make the prediction, and propose an optimal method to further smooth the obtained results with category neighbors and user neighbors. The evaluation is supplemented by the dataset from real world workload. The results show advantages of the proposed prediction approach compared with baseline prediction. In particular, the proposed approach can achieve 81.21% of Accuracy in gender prediction. While in dealing with a more challenging multi-class problem, the proposed approach can still achieve good performance (e.g., 73.84% of Accuracy in the prediction of age group and 66.42% of Accuracy in the prediction of phone level)

Improving Juvenile Age Estimation Based on Facial Landmark Points and Gravity Moment

Facial age estimation is of interest due to its potential to be applied in many real-life situations. However, recent age estimation efforts do not consider juveniles. Consequently, we introduce a juvenile age detection scheme called LaGMO, which focuses on the juvenile aging cues of facial shape and appearance. LaGMO is a combination of facial landmark points and Term Frequency Inverse Gravity Moment (TF-IGM). Inspired by the formation of words from morphemes, we obtained facial appearance features comprising facial shape and wrinkle texture and represented them as terms that described the age of the face. By leveraging the implicit ordinal relationship between the frequencies of the terms in the face, TF-IGM was used to compute the weights of the terms. From these weights, we built a matrix that corresponds to the possibilities of the face belonging to the age. Next, we reduced the reference matrix according to the juvenile age range (0–17 years) and avoided the exhaustive search through the entire training set. LaGMO detects the age by the projection of an unlabeled face image onto the reference matrix; the value of the projection depicts the higher probability of the image belonging to the age. With Mean Absolute Error (MAE) of 89% on the Face and Gesture Recognition Research Network (FG-NET) dataset, our proposal demonstrated superior performance in juvenile age estimation

A Software Digital Lock-In Amplifier Method with Automatic Frequency Estimation for Low SNR Multi-Frequency Signal

In the fault diagnosis field, the fault feature signal is weak and contaminated by the noise. The lock-in amplifier is a useful tool for weak signal detection. Aiming to the amplitude error of the lock-in amplifier caused by frequency deviation between the measured signal and the reference signal, a DFT-based automatic signal frequency estimation method is studied to improve the frequency accuracy of the reference signal. Based on this frequency estimation method, a software digital lock-in amplifier method is proposed to detect the multiple frequencies signals. This proposed method can automatically measure the frequency value of the measured signal without prior frequency information. Then, the reference signals are generated through this frequency value to make the digital lock-in amplifier estimate the amplitude of the measured signal. Moreover, an iterative structure is used to implement the multiple frequencies signal measurement. The frequencies and amplitudes measurement accuracies are tested. Under different SNR conditions, the frequency relative error is less than 0.1%. In addition, the amplitude relative error with different signal frequencies is less than 1.7% when the SNR is −1 dB. This proposed software digital lock-in amplifier method has a higher signal frequency tracking ability and amplitude measurement accuracy

Comparison of Ultrasound Type and Working Parameters on the Reduction of Four Higher Alcohols and the Main Phenolic Compounds

In this paper, studies were conducted by a series of single-factor experiments to investigate the effects of ultrasound types and working parameters on the higher alcohols (HA), phenolic compounds, and color properties of red wine, so as to highlight the importance of the comprehensive consideration on its application. The results indicate that ultrasound devices and working parameters do have some definite influences on the HA of wine; moreover, the ultrasound bath (SB-500DTY) is better than the SCIENTZ-950E and the KQ-300VDE. With the SB-500DTY employed to further investigate its effects on phenols and color properties other than on HA, unexpectedly, some variations of color parameters are opposite to the results ever obtained from other ultrasound conditions. In summary, all these results suggest that both the ultrasound type and parameters should be fully considered or neutralized so as to have a comprehensive evaluation about its application, instead of some contradictory results

Comparison of Ultrasound Type and Working Parameters on the Reduction of Four Higher Alcohols and the Main Phenolic Compounds

In this paper, studies were conducted by a series of single-factor experiments to investigate the effects of ultrasound types and working parameters on the higher alcohols (HA), phenolic compounds, and color properties of red wine, so as to highlight the importance of the comprehensive consideration on its application. The results indicate that ultrasound devices and working parameters do have some definite influences on the HA of wine; moreover, the ultrasound bath (SB-500DTY) is better than the SCIENTZ-950E and the KQ-300VDE. With the SB-500DTY employed to further investigate its effects on phenols and color properties other than on HA, unexpectedly, some variations of color parameters are opposite to the results ever obtained from other ultrasound conditions. In summary, all these results suggest that both the ultrasound type and parameters should be fully considered or neutralized so as to have a comprehensive evaluation about its application, instead of some contradictory results

A Software Digital Lock-In Amplifier Method with Automatic Frequency Estimation for Low SNR Multi-Frequency Signal

In the fault diagnosis field, the fault feature signal is weak and contaminated by the noise. The lock-in amplifier is a useful tool for weak signal detection. Aiming to the amplitude error of the lock-in amplifier caused by frequency deviation between the measured signal and the reference signal, a DFT-based automatic signal frequency estimation method is studied to improve the frequency accuracy of the reference signal. Based on this frequency estimation method, a software digital lock-in amplifier method is proposed to detect the multiple frequencies signals. This proposed method can automatically measure the frequency value of the measured signal without prior frequency information. Then, the reference signals are generated through this frequency value to make the digital lock-in amplifier estimate the amplitude of the measured signal. Moreover, an iterative structure is used to implement the multiple frequencies signal measurement. The frequencies and amplitudes measurement accuracies are tested. Under different SNR conditions, the frequency relative error is less than 0.1%. In addition, the amplitude relative error with different signal frequencies is less than 1.7% when the SNR is −1 dB. This proposed software digital lock-in amplifier method has a higher signal frequency tracking ability and amplitude measurement accuracy

Assembly and Capsid Expansion Mechanism of Bacteriophage P22 Revealed by High-Resolution Cryo-EM Structures

The formation of many double-stranded DNA viruses, such as herpesviruses and bacteriophages, begins with the scaffolding-protein-mediated assembly of the procapsid. Subsequently, the procapsid undergoes extensive structural rearrangement and expansion to become the mature capsid. Bacteriophage P22 is an established model system used to study virus maturation. Here, we report the cryo-electron microscopy structures of procapsid, empty procapsid, empty mature capsid, and mature capsid of phage P22 at resolutions of 2.6 Å, 3.9 Å, 2.8 Å, and 3.0 Å, respectively. The structure of the procapsid allowed us to build an accurate model of the coat protein gp5 and the C-terminal region of the scaffolding protein gp8. In addition, interactions among the gp5 subunits responsible for procapsid assembly and stabilization were identified. Two C-terminal α-helices of gp8 were observed to interact with the coat protein in the procapsid. The amino acid interactions between gp5 and gp8 in the procapsid were consistent with the results of previous biochemical studies involving mutant proteins. Our structures reveal hydrogen bonds and salt bridges between the gp5 subunits in the procapsid and the conformational changes of the gp5 domains involved in the closure of the local sixfold opening and a thinner capsid shell during capsid maturation

Assembly and Capsid Expansion Mechanism of Bacteriophage P22 Revealed by High-Resolution Cryo-EM Structures

The formation of many double-stranded DNA viruses, such as herpesviruses and bacteriophages, begins with the scaffolding-protein-mediated assembly of the procapsid. Subsequently, the procapsid undergoes extensive structural rearrangement and expansion to become the mature capsid. Bacteriophage P22 is an established model system used to study virus maturation. Here, we report the cryo-electron microscopy structures of procapsid, empty procapsid, empty mature capsid, and mature capsid of phage P22 at resolutions of 2.6 Å, 3.9 Å, 2.8 Å, and 3.0 Å, respectively. The structure of the procapsid allowed us to build an accurate model of the coat protein gp5 and the C-terminal region of the scaffolding protein gp8. In addition, interactions among the gp5 subunits responsible for procapsid assembly and stabilization were identified. Two C-terminal α-helices of gp8 were observed to interact with the coat protein in the procapsid. The amino acid interactions between gp5 and gp8 in the procapsid were consistent with the results of previous biochemical studies involving mutant proteins. Our structures reveal hydrogen bonds and salt bridges between the gp5 subunits in the procapsid and the conformational changes of the gp5 domains involved in the closure of the local sixfold opening and a thinner capsid shell during capsid maturation